Skip to content

feat: Offer --cache option to control cache mode (Incremental Build)#1368

Draft
maxreichmann wants to merge 223 commits intofeat/incremental-build-4from
feat/incremental-build-cache-mode
Draft

feat: Offer --cache option to control cache mode (Incremental Build)#1368
maxreichmann wants to merge 223 commits intofeat/incremental-build-4from
feat/incremental-build-cache-mode

Conversation

@maxreichmann
Copy link
Copy Markdown
Member

Part of Incremental Build: #1267

JIRA: CPOUI5FOUNDATION-1208

  • Introduce --cache with values "Default", "ReadOnly", "Force" and "Off" for commands "ui5 build" and "ui5 serve"
    • Add yargs help description (also appears in CLI docs)
  • Add JSDoc annotations
  • Rename --cache-mode to --snapshot-cache
    • Hide deprecated option in yargs help (if still used -> logs warning)
    • Legacy logic now usable via new name
    • Add hint under "Migrate to v5" in docs explaining this

TODO: Fix logic (+test) for multiple project using the same dependency (cache is expected to be reused)

RandomByte and others added 30 commits April 22, 2026 16:22
Cherry-picked from: SAP/ui5-fs@5651627
JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-builder@ef5a3b2
JIRA: CPOUI5FOUNDATION-1174
Prerequisite for versioning support

Cherry-picked from: SAP/ui5-project@83b5c4f
JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-cli@d29ead8
JIRA: CPOUI5FOUNDATION-1174
* Improve handling for concurrent resource access and modifications,
  especially when buffering streams.
* Deprecate getStatInfo in favor of dedicated getSize, isDirectory,
getLastModified methods.
* Deprecate synchronous getStream in favor of getStreamAsync and
  modifyStream, allowing for atomic modification of resource content
* Generate Resource hash using ssri
getIntegrity tests still need to be updated
Add unchangedNodes Set in TreeRegistry.flush() to track resource nodes
already confirmed unchanged via matchResourceMetadataStrict. Subsequent
trees sharing the same node skip the full comparison and only check tags,
eliminating N-1 async calls per shared resource across N trees.
Overlap stage cache I/O by prefetching metadata for the next task
while the current task is executing. This reduces idle time between
sequential task executions in warm-cache builds.
Add structured timing and counters to identify I/O hotspots in the
build cache validation flow. Instrumentation covers refreshIndices,
updateIndices, flushTreeChanges, TreeRegistry.flush phases, and
matchResourceMetadataStrict. All gated behind @ui5/logger perf level
or UI5_CACHE_PERF env var.
Add perf-level timing to the build orchestration layer to identify
the ~6.5s gap observed in warm-cache builds between source index
initialization and per-project cache validation. Instruments:
- getRequiredProjectContexts (total + per-project context creation)
- prepareProjectBuildAndValidateCache (getDependenciesReader vs cache)
- #flushPendingChanges (source index vs dependency index updates)
- ProjectBuilder.#build per-project timing
Defer source index initialization from ProjectBuildCache.create() to a
separate initSourceIndex() method, allowing BuildContext to initialize
all project source indices concurrently via Promise.all instead of
sequentially in the dependency discovery loop.
During dependency index updates, the cache proxy reader eagerly called
cacache.get.info() for every byPath() request, even though the callers
(hash tree upserts) only need resource metadata. With 2550+ parallel
calls, this created severe filesystem I/O contention (~2.2s).

Defer the cache path resolution to first content access using a lazy
singleton promise pattern. Also replace O(n) Array.includes() with O(1)
object property lookup and fix the size/byteSize parameter mismatch.

sap.ui.layout updateDependencyIndices: 2466ms -> 59ms
Total warm-cache build: 2.85s -> 315ms
…ing cache writes

Track integrity hashes from restored stage metadata in a Set and skip
cacache.get.info() calls for resources already known to be in CAS.
Reduces cache write time from ~1,400ms to ~100ms for stale-cache builds
by eliminating ~15,000 redundant CAS existence checks.
… import

On first CLI invocation, #importStages treated all restored stage
resources as "changed" because #currentStageSignatures was empty.
This caused ~3000+ resource paths to be propagated to dependents,
triggering expensive updateDependencyIndices calls (~108ms total).

The imported stages represent the already-cached state, not actual
changes. Skip writtenResourcePaths accumulation when this is the
initial import (empty #currentStageSignatures), since dependents'
dependency indices were restored from the same persistent cache.
…pagated

When restoring from cache, dependency indices are already populated
via BuildTaskCache.fromCache. If no dependency changes were propagated
from upstream projects, the cached indices are already correct and
_refreshDependencyIndices can be skipped entirely. This avoids
fetching ~2738 resources per dependent just to confirm nothing changed,
saving ~130ms on warm-cache builds.
…esult, and source freeze

Add perf-level timing to previously unlogged operations that dominate
stale-cache build time:

- allTasksCompleted: overall timing + sub-timings for
  revalidateSourceIndex and freezeUntransformedSources
- revalidateSourceIndex: byGlob timing for source file re-read
- freezeUntransformedSources: byPath reads and writeStageResources
- recordTaskResult: overall timing + delta merge and recordRequests

Investigation of a stale-cache sap.m build (1 file changed) showed
allTasksCompleted taking 11.3s of a 12s build, with
freezeUntransformedSources accounting for 10.2s due to per-resource
CAS existence checks.
In #freezeUntransformedSources, retain the previous build's frozen source
resourceMetadata and reuse entries for untransformed paths that haven't
changed, avoiding byPath reads and metadata collection for ~12K resources.

For sap.m stale-cache builds (1 file changed), this reduces
#freezeUntransformedSources from ~1,212ms to ~31ms.
Cover the RESTORING_DEPENDENCY_INDICES state handling in
prepareProjectBuildAndValidateCache, verifying that:

- _refreshDependencyIndices is skipped when no dependency changes
  were propagated (warm cache scenario)
- dependencyResourcesChanged() moves state to REQUIRES_UPDATE,
  routing changes through #flushPendingChanges instead
- State correctly transitions from RESTORING_DEPENDENCY_INDICES
  to FRESH after the first prepareProjectBuildAndValidateCache call
- Subsequent dependency changes use the REQUIRES_UPDATE path
Add skill with architecture reference covering class hierarchy,
Resource content model, adapter internals, collection patterns,
and resourceFactory API.
Replace cacache's content-addressable storage with a lightweight custom
implementation that computes content paths synchronously from integrity
hashes, eliminating ~5 seconds of cacache.get.info() index lookups per
build when writing ~14k resources to dist.

Key changes:
- New ContentAddressableStorage class with synchronous contentPath()
  computation, gzip-compressed storage, and atomic writes
- CacheManager now delegates to CAS module instead of cacache
- ProjectBuildCache resolves CAS paths synchronously (no PassThrough
  bridge needed in createStream factory)
- CAS-backed resources get sourceMetadata {adapter: "CAS"} to prepare
  for future write-path optimizations
- Bump CACHE_VERSION to v0_3 (breaking: old caches are invalidated)

BREAKING CHANGE: Build cache version bumped from v0_2 to v0_3.
Existing build caches will be rebuilt on first run.
Skip the PassThrough intermediary stream when writing CAS-backed resources
to disk. Instead, pipe the resource stream directly to the write stream,
eliminating one pipe hop and one stream object per resource while maintaining
stream backpressure for I/O throttling at scale.
Cover the previously untested delta merge logic (lines 798-848) that
executes when recordTaskResult receives a cacheInfo object from a delta
cache hit. Five new tests verify resource merging, tag import, tag merge
precedence, signature passthrough, and getCachedWriter fallback.
Introduce MetadataStorage class backed by node:sqlite (DatabaseSync)
to replace individual JSON file reads/writes in CacheManager. All four
metadata types (index cache, stage metadata, task metadata, result
metadata) are stored as JSON blobs in a single SQLite database with
composite primary keys, WAL mode, and prepared statements.

CacheManager remains the public interface, delegating metadata
operations to MetadataStorage and binary content to CAS. Unused
readBuildManifest/writeBuildManifest methods removed. CACHE_VERSION
bumped to v0_4.
Store gzip-compressed resource content as BLOBs in a SQLite database
(content.db) instead of individual files in a directory tree. This
eliminates per-file overhead (access, mkdir, writeFile+rename) and
enables batch writes within a single transaction.

Benchmarks on sap.m (11,893 untransformed sources):
- Cold cache writeStageResources: 4605ms → 2915ms (-37%)
- Cold cache total build: 25s → 22s (-12%)
- Warm cache: unchanged (~35ms)

Bumps cache version to v0_5.
Wrap metadata writes in explicit SQLite transactions (BEGIN/COMMIT)
to reduce per-statement WAL sync overhead, mirroring the existing
batch pattern in ContentAddressableStorageSQLite. ProjectBuildCache
.writeCache() now wraps all metadata writes in a single transaction
with automatic rollback on error.
Replace ContentAddressableStorageSQLite and MetadataStorage with a single
BuildCacheStorage class backed by one SQLite database (cache.db). Remove
the dead file-based ContentAddressableStorage.

Content batches use SAVEPOINTs when nested inside metadata batches,
preserving the existing independent rollback semantics with a single DB
connection. Bump CACHE_VERSION to v0_6.
@maxreichmann maxreichmann force-pushed the feat/incremental-build-cache-mode branch from 13173d5 to 68a130f Compare April 29, 2026 07:39
@maxreichmann maxreichmann marked this pull request as draft April 29, 2026 07:42
@maxreichmann maxreichmann force-pushed the feat/incremental-build-cache-mode branch from 68a130f to 0c087f8 Compare April 29, 2026 08:23
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from e1d781f to ab65f27 Compare April 30, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants